Vision based advanced driver assistance systems (ADAS) extract knowledge about the surrounding traffic scene by object detection and tracking. Typical objects to detect include other vehicles, lane markers, traffic signs, pedestrians, and others. A typical image processing system may comprise the modules feature extraction, object detection, tracking, and classification as well as the ADAS function/application itself. A feature extraction module analyses low level properties of the captured frame like edges, corners, intensity histograms, and others. Optionally optical flow estimation may be applied to gather information on the temporal behaviour of the scene. Stereo camera systems are a possible technology for future vision based ADAS. Depth information of a scene can be obtained by estimating the disparity of a pair of images.
Furthermore, object tracking is a known technique to gain knowledge about the temporal behaviour of detected objects and to support object classification. For example, Kalman-filters or Extended-Kalman-filters are widely used for tracking.
Correlation techniques on image regions and/or image features are used by known object tracking algorithms to find objects in subsequent images of a video stream. For these approaches it is crucial that a certain object has a similar appearance on subsequent image frames. In case of local image distortions this requirement may not be fulfilled and hence, these algorithms may “lose” the object of interest. Local image distortions can be caused by raindrops on the windshield, contamination of windshield, stone-chippings, and other disturbed areas on the windshield.
Effects of bad weather (e.g. raindrops on the windshield), contaminated areas as well as small damages (stone-chippings) lead to several types of image disturbances like (partial) occlusion of image regions, blur, and local distortions. As a consequence feature extraction performance will deteriorate as well as the detection performance of following modules.
JP Patent Laid Open Publication No. 2005-346387 (Patent Document 1: Application No. 2004-164992) describes a method for detecting moving objects with a camera mounted on a moving vehicle. In Patent Document 1, “noise” is mentioned as a disturbance which can be eliminated by this prior invention. However, according to the disclosure of this document, the prior invention deals with disturbances caused by bumpy roads. Therefore the “noise” mentioned in this patent document is a “pitching” of the vehicle which is compensated in an ingenious way. However, this method can hardly be applied to image restoration in rainy weather because there will be raindrops on almost every image and as a result the latency time for image processing will increase due to the need to wait for the next image frame or an image frame without noise.
JP Patent Laid Open Publication No. 2007-074326 (Patent Document 2: Application No. 2005-258667) describes a method for tracking moving objects which are temporarily optically occluded by an obstacle. Furthermore a warning method for such temporarily occluded objects is described.
PCT/JP2009/000181 filed Jan. 20, 2009 (Patent Document 3) and published Jul. 29, 2010 as WO 2010/084521 discloses a method for detecting raindrops on a windshield by comparing the image formed by each suspected spot on the windshield with an external view by applying a certain coordinate conversion to one of them and evaluating the correlation between the two images.
Meanwhile, there have been efforts to identify the presence of raindrops on a windshield for the purpose of automatically activating a windshield wiper. The previous proposals were mostly based on an electric sensor whose resistivity changes when raindrops deposit on the sensor. This can be achieved at a relatively low cost, but is not highly reliable because the surface of the sensor could be contaminated over time.
More recently, there has been a growing interest in the use of a vehicle vision system for tracking a lane marker or a center line, detecting an obstruction on the road, detecting a pedestrian and other purposes. Such a vision system can be conveniently used for detecting the presence of raindrops on a windshield. However, due to several factors, the detection of raindrops on windshields is a challenging computer vision task:                1. Raindrops have a large variety of different shapes and sizes.        2. Raindrops are blurred since they are out of focus (the camera is focusing on the traffic scene and not on the windshield).        3. Due to the transparency of the windshield, the observed raindrops are superimposed by interfering background information (for normal driver assistance systems, it can be defined vice-versa: the superimposition of raindrops is interfering the observed traffic scene).        4. Raindrops themselves are transparent, i.e., there are no defined features that are characteristic for raindrops. Raindrops rather reflect characteristic points from the environmental traffic scene.        
Japanese patent laid open publication (kokai) No. 10-148681 discloses a raindrop detecting method which detects a raindrop on a windshield as an area demonstrating a higher luminance than the surrounding areas. This however may not work as desired when the general view from the vehicle includes areas of high luminances or bright spots caused by emission and/or reflection of light.
Japanese patent laid open publication (kokai) No. 9-142259 discloses a raindrop detecting device which detects a raindrop on a windshield as disturbances in the transmission of light through the windshield. This device however requires a light emitter and a light receiver dedicated for the detection of raindrops. Therefore, it has the disadvantage of requiring added expenses and reserving suitable mounting spaces for the light emitter and light receiver in a limited available space of a vehicle.